Skip to content

Conversation

@vincentmacri
Copy link
Member

@vincentmacri vincentmacri commented Dec 4, 2025

Also fix a few rules that were only violated a small number of times.

This PR:

  • Removes list of rules we want to follow from src/tox.ini to remove duplication
  • Defines all rules we wish to follow in pyproject.toml
    • For rulesets that are not stable enough yet (E, PL) we list the rules explicitly instead of enabling all E or all PL rules.
    • We still ignore specific E and PL rules that we don't want to follow because the hope is that once the E and PL rules are fully stable in ruff we will just enable them all and selectively disable the ones we don't want.
  • We add a new file .github/workflows/ruff.toml that starts with the pyproject.toml configuration (using extend, so no duplication), and disables all rules that currently fail.
  • We also ignore a few troublesome directories in the ruff config, and run ruff on the entire repo. This means tools and scripts are linted now as well.
  • I also fixed a handful of rules that were only violated in a couple files, I didn't want to add a rule to the CI ignore list if fixing the rule was a one line change.

The benefits of this are:

  • There is now only one file (pyproject.toml) that needs to be modified to add new rules
  • We can now locally run the same ruleset as CI with ruff --config .github/workflows/ruff.toml

📝 Checklist

  • The title is concise and informative.
  • The description explains in detail what this PR is about.
  • I have linked a relevant issue or discussion.
  • I have created tests covering the changes.
  • I have updated the documentation and checked the documentation preview.

⌛ Dependencies

@vincentmacri vincentmacri added github actions Pull requests that update GitHub Actions code t: refactoring labels Dec 4, 2025
Copy link
Contributor

@tobiasdiez tobiasdiez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot, this is a nice improvement. Especially the handling of the rules activated in CI is way cleaner now.

I have a couple of remarks, but nothing serious.


# Imports at top-level of file - we can't follow this because we intentionally defer imports for various reasons.
# maybe once we can use PEP 810 this can be fixed.
"PLC0415",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there are actually quite a few imports in methods that could be moved to the top level without a problem. You want method-level imports only in one instance, namely to break circular imports, right? In that case it would be very helpful if the important is annotated to ease migrating to the new lazy imports (which doesn't help with circular imports) and so it's good if they have a comment like
# noqa: PLC0415 - breaks circular import x > y > z
Then during the migration to the built-in lazy imports, you know you don't need to touch those imports.

If it's just a heavy import that is rarely used in the module, it's okay to convert it to a lazy_import (and then later to the built-in lazy import).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In that case it would be very helpful if the important is annotated to ease migrating to the new lazy imports (which doesn't help with circular imports)

Lazy imports can help with circular imports depending on the situation. Sometimes a lazy import can fix a circular import, sometimes it can't.

As for what to do until we can use built-in lazy imports, there are so many linter failures related to imports that I think trying to address is futile unless we can reliably do it automatically, I don't think we can until built-in lazy imports are available.

If it's just a heavy import that is rarely used in the module, it's okay to convert it to a lazy_import (and then later to the built-in lazy import).

Yeah but I don't think it's worth the trouble of fixing thousands of lines for this, just to do it again once built-in lazy imports are available.

Copy link
Member Author

@vincentmacri vincentmacri Dec 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For context: PLC0415 currently has 10606 failures and has no autofix option available.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of weeks ago, I looked into how the migration to lazy import will most likely look like for us (and asked some clarifying questions to the PEP authors). Python will provide some way to call the underlying logic of the lazy import mechanism (similar to the dynamic import method). We can use this in our lazy_import method using something like

def lazy_import(module):
   if python >= 3.15: return python_builtin_lazy_import(module)
   else: our custom implementation

Then, once we have 3.15 as the minimal Python version (which will be in 4 to 5 years), we can replace lazy_import(...) by lazy import ... globally.
So it's a pretty long time frame until we can really profit from the new lazy imports syntax.

As for the local imports, the overall migration to lazy imports would look like:

  • Local import > global lazy_import
  • lazy_import > lazy import

Local imports that cannot be migrated to lazy imports, would need to be annotated in either case.

As for the reason why I like to keep PCL0415 activated is that IDEs highlight the import as a linter warning. So if you touch the surrounding code, you get a natural push to move the local import to a global (lazy) import. Then you usually run the doctests anyway, and will see if the import can be global or it really needs to be local (to break a circular import). I like to keep that nudge, as we would like to remove all possible local imports in the long term anyway.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So it's a pretty long time frame until we can really profit from the new lazy imports syntax.

Yeah but given the 10000+ failures of this rule currently, I think there are more important things that take less effort to fix with regards to our code quality.

As for the reason why I like to keep PCL0415 activated is that IDEs highlight the import as a linter warning. So if you touch the surrounding code, you get a natural push to move the local import to a global (lazy) import. Then you usually run the doctests anyway, and will see if the import can be global or it really needs to be local (to break a circular import). I like to keep that nudge, as we would like to remove all possible local imports in the long term anyway.

Sure. I can remove PCL0415 from the pyproject.toml ignore and move it to the CI ignore.

I think we can leave E402 disabled though, having a module-level import not at the top of the file is weird enough that I think any violations of the rule are probably just a case of mixing import and lazy_import statements. I think most of the violations of that rule are in places like all.py files that for whatever reason have a mix of regular and lazy imports.

Copy link
Member Author

@vincentmacri vincentmacri Dec 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. I can remove PCL0415 from the pyproject.toml ignore and move it to the CI ignore.

Before I do this: I don't think linters and type checkers can recognize imports from our custom lazy_import, can they? So wouldn't replacing function imports with lazy_import limit our ability to do linting and type checking?


# Module level import not at top of file.
# ruff is not aware of our lazy_imports, we may be able to remove this ignore after adopting PEP 810
"E402",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't quite understand the lazy_import argument. Can you not just have all normal imports at the top, then the lazy imports afterwards?

(We might want to defer this rule until there is a decision on how lazy import should be grouped/sorted - if they get a separate group anyway, that would be a good reason to already follow E402; if they are not grouped separately, then it would only introduce additional noise to first follow E402).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't quite understand the lazy_import argument. Can you not just have all normal imports at the top, then the lazy imports afterwards?

Sure, but I don't think it's worth the amount of effort to do that reformatting and then do it again if we decide to replace our custom lazy import system with the PEP 810 one.

@github-actions
Copy link

github-actions bot commented Dec 8, 2025

Documentation preview for this PR (built with commit c17ef62; changes) is ready! 🎉
This preview will update shortly after each push to this PR.

@vincentmacri
Copy link
Member Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

github actions Pull requests that update GitHub Actions code s: needs review t: refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants